Online curriculum handling system including content assembly from structured storage of reusable components

ABSTRACT

In a curricula system, courses are assembled from components where items are created by authors and/or authoring tools, items comprising questions, answers, narrative, media, interactions, or the like, and instructional designers design products that specify structures, strategies and the like for a product. A product, in the form of one or more course presentable to a student or other user, references reusable content items, such that a given item can be present in more than one unrelated product. A product can be represented with a hierarchical structure organized as one or more course each comprising one or more units, in turn comprising one or more modules, such as testing modules and learning modules, wherein each module in turn specifies items to be included. Products can take on one or more product template and one or more product class, wherein a product template specifies the “look-and-feel” of the product and the product class defines basic course structure and functionality. Items are atomic objects that are reusable across products, courses, units, etc. The atomic objects can be created in advance and can be stored in a non-product specific manner.

BACKGROUND OF THE INVENTION

The present invention relates to testing and learning systems in generaland in particular to testing and learning systems where components arereusable.

Testing and learning systems (generally referred to here as “curriculumsystems”) have been used in many environments. For example, teachersmight use them in the classroom to present material, test students, orboth. As another example, regulatory bodies might test applicants as aprecursor to granting a license (e.g., attorney exams, NASDqualification exams). As yet another example, schools or academicassociations might use tests as an indicator of student aptitude andpreparedness (e.g., SAT, MCAT, GRE, LSAT). Providers of testing andlearning services might often need to provide practice tests andcurricula for such tests. For example, a curriculum system might be usedfor preparing a student for taking a standardized test by giving thestudent practice questions, then simulating an actual test and, whereappropriate and possible for the testing topic, provide learning alongwith testing. For example, where a student is preparing for acontractor's exam, the curriculum system might provide sample tests andlessons in areas of a student's deficiency.

Where a provider of testing and learning services supports students inmany practice areas, the management of tests, lessons and othermaterials needed becomes difficult. In some cases, the processes can bemanaged well when the topics do not change very often, by publishingpaper materials that are copied for each student. However, where thematerial changes, such as reorganization of standardized tests, updatesto the topic (such as changes to what is covered in a particular exam,updates to the laws for legal/contracting/regulatory, etc. tests) andthe like occur often, or the students expect online access to thecurricula, simply printing one version of a course and reprinting itwill not be feasible. In addition, where each course is independentlyhandled, there would be much duplication as questions, narrative,images, and other elements are distributed over many different forms ofcontent. Therefore, improved systems and methods for handling elementsof curricula systems were needed.

BRIEF SUMMARY OF THE INVENTION

In one embodiment of curricula system, courses are assembled fromcomponents where items are created by authors and/or authoring tools,items comprising questions, answers, narrative, media, interactions, orthe like, and instructional designers design products that specifystructures, strategies and the like for a product. A product, in theform of one or more course presentable to a student or other user,references reusable content items, such that a given item can be presentin more than one unrelated product. In some embodiments, a product is ahierarchical structure organized as one or more course each comprisingone or more units, in turn comprising one or more modules, such astesting modules and learning modules, wherein each module in turnspecifies items to be included. In specific embodiments, products cantake on one or more product template and one or more product class,wherein a product template specifies the “look-and-feel” of the productand the product class defines basic course structure and functionality.Items are atomic objects that are reusable across products, courses,units, etc. The atomic objects can be created in advance and can bestored in a non-product specific manner.

The following detailed description together with the accompanyingdrawings will provide a better understanding of the nature andadvantages of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an online curriculum system according toone embodiment of the present invention; FIG. 1A is a high-level view;FIGS. 1B-1C show additional details.

FIG. 2 is a data diagram illustrating a data storage arrangement thatcan be used by the system shown in FIG. 1.

FIG. 3 shows more detail of the data diagram of FIG. 2; FIGS. 3A and 3Bshow different aspects thereof.

FIG. 4 is an illustration of a reference scheme.

FIG. 5 is an illustration of a search process.

FIG. 6 is a high-level representation of a search system.

FIG. 7 is a template mapping diagram.

FIG. 8 is an illustration of a Product Definition XML (PDX) fileexample.

FIG. 9 shows a process of content authoring.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 is a block diagram of a curriculum system 10 according toembodiments of the present invention. As used herein, curriculum refersto lessons, workshops, tutorials, activities, customized drill andpractice, pre-defined assessments, examinations, or the like. Acurriculum can comprise lessons for student education, or no lessons. Acurriculum can include one or more tests in a practice setting, asimulated test setting or an actual test setting. A curriculum isdirected at one or more students, wherein the students can beindividuals that seek to learn a subject, to identify their deficienciesin areas of knowledge, to test themselves on areas of knowledge, toprepare themselves for taking tests outside or inside the system, and/orrelated activities. A curriculum can have an identified study plan thatmight be linear and predefined, or prescriptive with one or manyiterations of prescription.

Using curriculum system 10, a curriculum administrator can create,manage and deliver interactive curriculum to students. As shown,curriculum system 10 includes authoring tools 20 coupled to a contentmanagement system (CMS) 30 coupled to a structured content storage (SCS)32. CMS 30 is also coupled to a product assembly interface 40 and acontent publishing system (CPS) 50. As shown, CPS 50 includes a directlink for accessing data in the SCS without going through CMS 30. Itshould be understood that other interactions, links and associations notexplicitly shown might exist, as a person of ordinary skill in the artwould understand. The CPS is shown coupled to an online learning andtesting platform (OLTP) 60 and a curriculum database (C-DB) 70. SCS 32might be an XML database or other structured storage and C-DB 70 mightbe an XML database, a hierarchical directory in a file store, acompressed structure of files, or the like. The OLTP is coupled to aperformance database 80 and a student database 82. Also shown arestudent interfaces to OLTP, such as by Internet access using a browseron a desktop computer or other computer, or via a mobile deviceinterface as might interface to a cellular telephone, a handheldcomputer, or other mobile device.

Curriculum system 10 can be a stand-alone system or integrated withexisting learning management systems to allow for the tracking ofstudents usage and progress through their study. Curriculum system 10provides curriculum authors with a set of authoring tools usable tocreate atomic instructional objects, including test questions, media andother objects. Referring now to FIG. 1B, authoring tools 20 mightcomprise an author user interface 22, automated content generators 26and input modules 28 for previously published content, such as books,CD-ROMs, articles, scanned papers, electronic articles, web pages, etc.

Authoring tools 20 allows for administrators and content creators tocreate objects elements. For example, an author might be provided with agraphical user interface (GUI) to an XML editor to allow for authoringcontent, including appropriate metatags used for assembly of products byproduct assembly interface 40 of CPS 50. The authoring tools might alsoprovide the ability to search for and/or edit content already stored byCMS 30 in SCS 32. Some of the metatags might be configured so thatquestion or lesson item content can be repurposed for online and/orprint uses, categorized within multiple curriculum and organizationaltaxonomies, and tracked for the protection of operator and/or authorintellectual property. For example, a question might include metatagsidentifying the question as a hard question, a math question, a finitealgebra question (being more specific in a taxonomy than the “math”metatag), as well as metatags identifying the author of the question andconcomitant intellectual property rights.

CMS 30 stores and manages content in a presentation-neutral format, suchas XML, structured text, SGML, HTML, RTF, or the like. CMS 30 also cantrack ongoing creation and modification of content using version controltechniques, as well as support access controls for intellectualproperty, user-management and security. CMS 30 might support the use ofthe proprietary authoring and search tools, the storage and deploymentof traditional curriculum, including simple to complex question types(e.g., multiple choice, picture marking, fill-in, line drawing, etc.) aswell as exact emulations of the layout and functionality of questions oncomputer based standardized tests (e.g. GRE, GMAT, SAT) and the itemsand structure can be independent.

CMS 30 can also be configured to store rich media assets includinggraphics, animations, and audio and video clips associated with questionand lesson content. Some of the functionality of CMS 30 might besupplied by off-the-shelf software. For example, content managementfunctions such as workflow, versioning, XML storage, Document TypeDefinition (DTD) editing for structured content storage, etc., might beprovided by a product such as Broadvision's One-to-One ContentManagement System. As shown, the data maintained by CMS 30 is stored instructured content storage (SCS) 32, but in some embodiments, CMS 30 andSCS 32 might be more integrated than is implied by FIG. 1.

Product assembly interface 40 allows an instructional designer to designa product, course, lesson, test, etc., from content in SCS 32. Productassembly interface 40 can be used to capture features a product shouldcontain, record these settings in a form CPS 50 can understand andidentify what instructional content will be included in a course ofstudy or testing. Thus, product assembly interface 40 can providestructure, strategies and hierarchies for a product or componentsthereof. The designer is often different from the author, as the authorscreate items and the designer builds a product from those items,specifying how it all comes together. However, nothing prevents the sameperson from being an author and a designer. One benefit of the systemshown in FIG. 1 is that both the author and the designer can benontechnical and provide input in an intuitive manner.

A typical assembly process comprises two sets of documents: (1) aProduct Definition Parameters (PDP) document that captures productfeatures and structure in a checklist fashion and (2) a PDX document,which is a more machine-readable version of the PDP. The PDX file isused by CPS 50 to enable automated publishing of curriculum and mediaassets from SCS 32 to OLTP 60, upon receipt of a publishing trigger. CPS50 can work with CMS 30, but in some cases, it might be more efficientfor CPS 50 to read directly from SCS 32. In some embodiments, OLTP 60includes designer inputs, to allow for automatic control of settings,such as the form of the output (HTML, XML, print, simplified for mobiledevices, etc.), as well as administrative rules and settings such aslook and feel settings, instructional design settings, etc.

If CPS 50 publishes a product in off-line form, the output can becamera-ready pages, PDF files or the like. If CPS publishes a product inon-line form, the curriculum is sent to C-DB 70, but some staticelements, such as media components, text, etc. are provided directly toOLTP 60. Some of those static elements might be stored on a fast Webserver storage unit for quick feeding to users as needed.

OLTP 60 can provide a broad array of online learning products usingcurriculum deployed from the CMS. The platform allows for the flexibleselection and utilization of learning components (e.g., tests,tutorials, explanations) when designing an online course. FIG. 1C showssome components of OLTP 60, such as product class and content templates62, a testing system 64, a reporting system 66 and a customizedcurriculum system 68.

Thus, the assembly interface can be used to provide structure andrelationships of the atomic elements informing the system of theirinstructional design strategies, and publishing tools can auto-generatecode and create a final product bundle to be delivered to the student ina context appropriate for their use. In a specific implementation, C-DB70 is an Oracle database and OLTP 60 includes an interface to thatOracle database, an interface to middleware such as Weblogic's productand a Web server interface.

Content Management System

Once files are created as shown in FIG. 9 (below) or by other methods,they are stored in structured form into SCS 32 by CMS 30. One contentmanagement system that could be used is Broadvision's One-to-One Contentsystem. Such documents could be stored as XML documents generated byKaplan's authoring system and automated parsing tools. In oneembodiment, XML documents are stored in a repository with a project anddirectory metaphor. As used herein, the term “item” is used to refer toobjects stored by CMS 30 as atomic units. In many products, each item ispresented to the student separately, such as by clearing a screen andusing the entire screen to present the item, without other items beingpresent.

Preferably, items are stored by CMS 30 using globally unique identifiers(GUIDs). When a product is created, a list of items, identified by theirGUID, can be created. The CPS extracts the items from CMS 30 accordingto this list and compiles them for use in the specific product. In thisway, any single content item may be referenced by several products withno further modifications or editing required.

As an example, a product might be a particular test for a particularmarket and set of students. If the test contained 1000 questions, invarious places, the list for that product would reference thosequestions in the CMS by their GUIDs. One advantage of this approach isthat questions can be authored and stored separately, then labeled inthe CMS using a contextually neutral GUID. The questions do not need tobe aggregated for use in the product until the time of publishing theproduct, and the questions can be reused easily from product to productand can be updated in one place and have the updates propagatedthroughout all new and republished products.

In order to find items easily and according to specific productrequirements (e.g., every “hard” math question, etc.), items mightfurther include associated metadata that describes the content in aproduct-neutral manner. Thus, general taxonomies may be used to organizeitems before they are placed in specific products.

Platform Data Model

The data stored in the CMS can be structured according to the platformdata model described herein. The platform data model is optimized forthe re-use of content. A referential document model fulfills thisobjective, where atomic units of content (items), such as questions,media, lesson pages, glossary words, etc., are provided GUIDs.

In addition to items, the CMS might also track products and references.Thus, in the basic system, there are three classes of data: content,products and references. Content includes questions, media and othercontent, without requiring any specific product-contextual information,which is preferably absent to allow for easy reuse. Product dataincludes product item, product delivery rules, PDX files, etc.,containing product-specific information about referenced content itemsor product items, including categories, difficulty levels, userinterface display instructions, rules to be applied to referencedcontent, etc. Referential data includes pointers between items andproducts and/or items and items (and possibly even products toproducts).

FIG. 2 illustrates an example of data structures according to theplatform data model, showing productItem records,productItemDeliveryRules records, item records, category records,content records, question records, media asset records, and the like.FIG. 2A illustrates an example of XML document types according to theplatform data model, showing productItem.xml,productItemDeliveryRules.xml, item.xml, category.xml, content.xml,question.xml, mediaAsset.xml, and the like. These document types containx-link references that determine their relationship to other documenttypes.

FIG. 2B shows one possible structure for data defining the hierarchy ofa product, such as courses, units and modules. For example, referencesto a number of items might be grouped to form a lesson module and otherreferences grouped to form a test module. These modules can be groupedinto a unit and one or more units would comprise a course. Each coursecan be a product, but a product might also comprise multiple courses. Asused herein, “plannable component” refers to one of the building blocksof products, including units, lessons, tests, tutorials, references,tools and the like. In particular embodiments, these are the buildingblocks available to a designer, so that any block a designer can selector unselect for a product would be a “plannable component”. A productmust have at least one plannable component, but there need not be alimit to the number of components a product can have. Each plannablecomponent has a unique set of properties and functionality that is usedto customize its operation within a course. These plannable componentsend up being identified as such in the product definition file(s) forthe product.

FIG. 3A illustrates the structures of the data model that might be usedfor authoring a simple text-only question, such as an “analogy”question. FIG. 3B illustrates the structures of the data model thatmight be used for a data interpretation question-set. As shown there, aproductItem record has a category and an item, which in turn has aproductItemDeliveryRules record. The item record relates to a set ofquestions, media assets and other content, such as a stimulus diagramand a question-set explanation. Both content and question files can linkto a reference file.

A reference file is based on a reference schema, such as the one shownin FIG. 4. In that schema, the root element of the reference schema is<referenceDefinition>. The element <referenceDefinition> contains thename of the reference and the name of the set the reference belongs to,but it does not contain any of the text/images of the reference itself.For this, it links to one or more content files.

Presentation-Neutral Item Structure

While the document-centric nature of the platform data model supportsre-use, the use of presentation-neutral constructs within each documenttype further supports the ability to abstract pure content from how itmight be realized in a particular product. For example, the followingsentence could be part of a question item:

The book, “Tom Thumb” is about a fictional character of the 18^(th)century.

The XML-encoded version of this sentence might be:

-   -   <matinline>The book, <bookTitle>Tom Thumb </bookTitle> is about        a fictional character of the 18^(th) century.</matinline>

By using the term “bookTitle” to describe a particular type of phrase orterm, the actual visual presentation of “Tom Thumb” could be realized inbold, underline, etc., according to the demands of a specific product.Each product description document (see below) contains a set ofpreferences that can be unique that map these presentation requirementsto the actual product.

One advantage of using a presentation-neutral item structure is that thedetails of test strategy, presentation, look-and-feel can all beseparated from the items that will be used in a product, thus allowingitems to be created once and course plans created once, with each ofthose being reusable and relations between which items are in whichcourses to be flexibly applied. Furthermore, where the items and courseplans are provided in a structured form, they can be edited by possiblynontechnical users. This would allow, for example, a designer to designa new course from previously used content and/or new content, with avarying presentation and structure, all without having to reprogram thesystem (such as OLTP 60 or CPS 50) that presents or publishes thecourse. Thus, a product could be created “on the fly” as a designerselects templates and content and those selections are stored in SCS 32.

The structure for item storage described herein also allows for easyupdates. For example, if the answer to a question changes (“Who is thecurrent President of the United States”?), the change only has to bemade to the question items that change. When a course is republished, itwill be again constructed from the items and the PDX files and theanswers will appear updated.

Because both the content and the structure of a course can be easilychanged, a course designer could easily vary strategies to determinewhich strategies allow students to learn better. Where the course ispublished online, the course designer could vary the strategies on avery fine schedule to quickly fine tune the process. This fine tuningmight be part of a feedback system wherein students take tests, theirperformance is monitored (e.g., right answers, time delay betweeninteractions, help used, etc.) and those results are used to rankdifferent strategies so that the optimum strategies can be used.

With the tools described herein, the variations of items, strategy andother elements of a course can be created and manipulated by editorialstaff instead of requiring programmers and other technical staff, thusallowing the course creation by those closer to the educational process.Where only one product or course is being created, this is not an issue,but it becomes a significant issue where many courses, in many areas,are to be created and administered.

CMS Search Engine

The unique, referential nature of the platform data model can be easilysearched using the search engine described here. The search engine canintelligently negotiate the references and find individual items in thecontext of their various parent and child relationships. The CMS searchengine extracts individual XML items in the repository, transforms themto a searchable view, casting off elements that are not required forsearch, resolves the references and then maps the data to a series ofdatabase tables. This search engine might be accessible to authors viaauthoring tools 20 and to designers via product assembly interface 40.

FIG. 5 illustrates an example of a search as might be performed by theCMS search engine. Suppose a user needs to find all products that use amedia object named “triangleABC.gif.” Following are the logical stepsfor carrying out this search, as shown in FIG. 5:

-   -   1) Find the media object triangleABC.gif and verify its        existence in the repository.    -   2) Find any content or question that contains a reference to        triangleABC.gif.    -   3) Find any product items that refer to the contents or        questions found in Step (2).    -   4) Find any plannable components that refer to the product items        found in Step (3).    -   5) Find any PDX files that refer to the plannable components        found in Step (4).

The searchable view component of the search engine allows for resolutionand storage of these relationships before insertion into the searchdatabase, thus pre-empting the need to actually traverse the items inthe course of a search.

FIG. 6 is a high-level visual representation of the search system. Thesearch system extracts new files from the repository and inserts theupdated information into the database on a periodic basis. The XMLMapping mechanism is modular in the sense that if a new schema iscreated, only the mapping needs to be adjusted to match the new schema.Underlying processes automatically update or re-format the database tomatch the new data model.

In one embodiment, the search engine is built as a set of Java classesthat are exposed to developers as a toolkit accessed by Java APIs.Developers can then build any user interface above this toolkit andaccess the functions of the toolkit via the APIs.

Product Assembly Interface

Product assembly interface 40 provides a method for applying productlevel parameters to content that will be assembled into a product andincludes a set of tools and processes used to record and communicate theproduct settings to content publishing/delivery system (CPS) 50, usuallyvia SCS 32. Product assembly interface 40 captures information on theproduct structure and operation. Preferably, all assembly informationcan be recorded into a series of XML files and Product Definition XML(PDX) files, such as the examples shown herein.

The PDX files reference content and media to be used within the product,directly or via “indirect” file references. Such information includescategory definitions, definitions of which user interface files to useon particular categories of content, definitions of what rules will beapplied to certain categories of content, such as gating and evaluation,variable help and introductory copy. Other information might beincluded, such as references to every items used in the product (andindirectly every content question and media item), as well as componentnames and test names and rules.

The PDX files might also include indications of course strategy. Forexample, a course's specification might include reference to pluggablecomponents of code and/or rules used by the OLTP to control variousaspects of the curriculum and user experience. Examples of suchinteractions include, but are not limited to, item selection, next item,performance calculation, question evaluation, scoring, sectioncompletion, section passing, test completion, termination, coursecontrol, achievement criterion, parameter validation and study planner.

Content Reuse

The system supports at the following content reuse scenarios, as well asothers that should be apparent from this disclosure. The first isselecting specific content units for use in other products; contentwould remain unchanged and inherit any changes made to the source file.Another scenario is reuse subset, wherein the system supports a subsetof content reuse, i.e., content copying. Authors will select a contentunit or an individual file and make a copy of it for use in anotherproduct with no links made back to the original source file. The copywill receive a new identifier (GUID, RID, QID, etc.).

The Global Unique Identifier (GUID) is a number is generated usingalgorithms ensuring that it is globally unique. Resource Identifiers(RID) or Object Identifiers (OIDs) are IDs assigned to identify an item.These IDs may or may not be unique and are managed by the system thatassigns the RID. Question Identifiers (QIDs) are unique IDs within thescope of the platform, typically displayed to the customer or otherend-user, used to identify a piece of instructional content duringservice calls.

Some actions performed by product assembly interface 40 will now bedescribed. When a designer is specifying a product, the designerspecifies a product class and product template that the product willuse. Selection of the product class determines structure of the PDX andcomponents usable within the product. The product template determinesthe product's UI (user interface) and content organization. In someembodiments, the product assembly interface enforces product class andproduct line selection prior to allowing the designer to proceed withproduct creation. A product's class determines a specific structure ofthe PDX and the component(s) used within the product.

Examples of product classes are shown in FIG. 7. Products within thesame class, regardless of content, share the same basic structure andfunctionality. A product's line determines the presentation of theproduct, including UI color scheme, look-and-feel, content taxonomy, howto present questions on the screen, etc. Selecting the product line willset values within the PDX corresponding to the user's selection.Typically, the product line represents a set of test, instructions,materials and/or offerings that have a common market segment. Forexample, one product line setting could be for the GRE, another for theLSAT, another for the GMAT, etc.

Based on the product class selected, a list of product components willbe presented to the designer. The designer will create the coursestructure by indicating which component to use along with order and namefor component. Course structure might include the type of productcomponent and a sequence in relation to other components at the samelevel within the course. A PDX file might exist for each product classand the product classes and product lines are preferably editable forease of making changes. With a moderately sized set of product classesand product lines, the designer might be presented with a matrixinterface essentially like the table shown in FIG. 7 and be allowed toselect one or more cells of the matrix to define the product class(es)and product line(s) for a product.

The product assembly interface 40 enforces component use rules dealingwith acceptable component hierarchy (e.g., option of having lesson pageslimited to being added to lesson components) and required unique entries(component names). Content validation or pedagogic validation need notbe performed. The designer can modify a course structure at any timeduring product creation, but components selected by the designer, alongwith sequencing information, will be written to the PDX prior toallowing other user actions.

As part of product assembly, the course designers choose thepresentation templates that will map to a product. Authors user elementsof the system to create items and lesson content, while course designersdesign the pedagogy and flow of a course/product. In addition toselecting product classes and lines, the designer might also specifywhich content to use and add strategies, reports and the like, to theproduct. In some cases, someone can be both an author and a coursedesigner, but the system allows for separate specialties to be usedeasily.

Examples of Presentation Template Functions Include:

1) Template Assignments: ID/CDs assign products (from a course, unit,lesson or individual page basis) to platform presentation templatesusing a WYSIWYG tool. This includes general templates (for quizzes,activities, tests) and specific templates (for particular lesson pageconfigurations, such as content with a left sidebar, content with nosidebar, etc.). Templates are chosen from a library of predefinedplatform templates.

2) Course Parameters: Based on the Class, parameters are presented tothe designer for setting course/component operation and allowableassembly operations. The line selected by the designer determines theoptions available for each parameter. Two groupings of parameters thatmight be presented to designers are product parameters and assemblyparameters.

Product parameters set how the product will perform. The values areentered into the PDX. Assembly parameters specify how the assembly toolswill interact with the product being created and define allowableaction. The selections made by the designer are not required to bewritten to the PDX, but should be stored for use while designers arecreating the product.

The instructional items used within the product have parameters set thatimpact product performance and how the content is handled in therepository. Similar to the course structure requirement for sequencingof components, instructional items have a parameter set that determinesits sequence among all items within the component. Categories are ataxonomy used to organize the content for organization, reporting, andpresentation within the platform and product. The product line definesthe acceptable categories for use within a product. The designer selectsone or multiple categories, from predefined lists, to assign to theitem.

Product Definition Parameters File

All of the product definition parameters can be stored in a PDP file ina format such as that shown in Appendix A as Table A. 1. It should beunderstood that the specific format shown is not required and otherformats might be used just as well. For example, the PDP might bepresented as a set of checkboxes to be filled in.

In actual storage, the product definitions would be in a more“machine-readable” form, such as a Product Definition XML (PDX) file (orfiles) as illustrated by the example of FIG. 8. A PDP file might becreated using a checklist provided to the designer through the productassembly interface 40.

From a completed PDP, the PDX documents can be created. The PDX are aset of documents that capture the product features and curriculumstructure in a form that can be understood by CPS 50. The parametersdocumented in the PDP are converted to a structured XML format, andacceptable settings that the CPS will use to create the product. The PDXdocument structure, while a unique format used to instruct the CPS, canvary by class of product and structure of the course.

From the PDX, the CPS can determine information needed for packaging aproduct for publication, such as 1) uniquely identifying the course(s)being created, 2) the course parameters defined in the PDP in a machinereadable form, 3) the relationships between all components of the course(units, lessons, tests, deliverable pages, etc.), 4) references to allcurricular content to be used in the course, and 5) the rules the OLTPwill use for presentation, course navigation, and evaluation of thestudent's interaction with the course. The CPS interprets theseinstructions during transformation of the course content into adeployable OLTP course.

Based on the product class, a specific list of features and options areavailable within the PDP. The product design and feature set is createdas the designer selects from predefined options for each feature. Theoptions are textual descriptions of the expected functionality for aspecific feature. When completed, the PDP provides a detaileddescription of the products expected functionality and performance in“human-readable” form.

The completed PDX describes a complete and unique product. CPS 50 canread the PDX to learn what instructional content to include and how itshould be presented and from that generated a product where the contentand instruction on how the product should perform within the platform(such as how it interacts with its users if it is an online product, orhow it looks on the page if it is a printed product) are packaged withina single unique deployable package.

Content Publishing System

The CPS is coded to interpret information within the PDX files and tocompile the reference instructional content and instructional rules intoa finished product. The CPS extracts all of the data related to a singlecourse as defined within a specific PDX document [examples shown in FIG.8] contained within the CMS. References to curriculum components, suchas those shown in the structures of FIGS. 2-4 and 8 and might includetest questions, lesson pages, media assets and strategies, are resolvedto the actual implemented components contained elsewhere within the CMSand SCS 32. The data is then transformed and packaged for finaldelivery. In the case of curriculum to be delivered online, the OLTP 60extracts the package and inserts it into C-DB 70 for future delivery orthe CPS provides it to C-DB 70.

Online Learning and Testing Platform (OLTP)

One of the publishing routes is to publish to an online learning/testingplatform (OLTP) 60 that provides products in online form. Within theOLTP, products designed by course designers include online delivery ofcurriculum (including tests and assessments, explanations and feedback,lessons and customized content) to customers and this might be done viaa standard Web browsers and Web protocols.

OLTP 60 can also generate reports on student performance and to providecustom interpretation a student can use for future test preparation andstudy planning as well as deliver to students functionality for thestudent to self select learning modules or to have the platformautomatically prescribe customized curriculum based on assessmentresults and student entered preference information.

Delivered products can be used in a self-study mode, including (1)simple single topic linear tests, multi-sectioned tests with scaledscores, or student customizable practice tests, (2) diagnosticassessments with simple score reports or diagnostics providing richnarrative feedback and recommended study plans, and (3) complete courseswith tests and lesson tutorials delivered in a simple linear pedagogy orindividualized courses, customized to meet unique student learningneeds.

The OLTP provides the designer with the choice between working from apre-set structure defining a particular product class or to selectsubcomponents that compromise an existing structure to create newproduct classes. Three examples of pre-defined templates used to definea product in the OLTP are the product class templates, the productbranding templates and content interface templates. Product classtemplates might be:

-   -   1. Student Customized Test    -   2. Continuing Education Course with Linear Tests    -   3. Student Customized Test with Full Length Linear Test    -   4. Full Course (Student Customized Test, Full Length Linear Test        and Course Material)    -   5. Multi-Section Exam    -   6. Computer Assisted Feedback    -   7. Course with Localized Content for Institutions    -   8. Course with Prescriptive Study Plan

Where the designer can select product classes, such as by selectingcells in the matrix shown in FIG. 7, the designer might select multipleproduct classes and product branding templates. Product brandingtemplates might provide a particular provider's look-and-feel oremulation thereof, such as:

-   -   1. Financial Best Practices Interface    -   2. Real Estate Best Practices Interface    -   3. Kaplan Test Prep Best Practices Interface        -   a. K-12 Achievement Planner        -   b. Generic Test Prep    -   4. Testing Service Emulation    -   5. Other

Content interface templates might include question type templates,response type templates, lesson interface templates and the like.

Testing System

The testing system supports online and offline administration of tests.Tests can be defined as a series of questions grouped and administeredin a variety of interfaces that can be presented in numerous formatsincluding short practice quizzes, sectionalized tests and full-scalestandardized test simulations and review or practice. The studentinteracts with this content in various ways, while the system tracksdata about these interactions: answers chosen, time spent, essay text,and more. For tests administrated offline, the testing system canreceive the data through a proxy. The system supports a variety of itemadministration rules, including linear, random, student selected(custom), and adaptive testing. It also supports rules governing the waya test instance is presented to the user (e.g., test directions, help,breaks, etc.). The testing system might specify or control the followingaspects of a test process:

-   -   A. Ability to define passing criteria per test    -   B. Ability to define timing by test, by category, by item    -   C. Ability to define test class, e.g., pre-test, post-test, and        organize reports based on test class    -   D. Ability to define recommendation level, e.g., required,        optional    -   E. Ability to reuse items across tests and across products    -   F. Ability to define secure items that can appear in a given        test or product only, e.g., a final exam    -   G. Ability to develop tests that emulate the standardized        computer based tests including multi-section administration,        scaled scoring and adaptive delivery

Many different types of tests can be accommodated by the testing system,with potentially unlimited number of tests of any type per course. Testtypes can be mixed and matched. For example:

-   -   1. Customizable tests (Qbank and Drill and Practice)        -   a) Ability to create any number of custom tests (based on            reuse, difficulty level, category) from a single test            definition.        -   b) Ability to create multiple Custom Test “factories” or            test definitions in a single product, e.g. you can define            subject-specific Custom Tests for individual Units and a            comprehensive Custom Test that covers all course material.    -   2. Predefined linear and multi-section tests    -   3. System-generated linear tests with the following variations:        -   a) Ability to generate new test with shuffled (but otherwise            same) set of items        -   b) Ability to generate new test with fresh selection of            items based on item selection rules defined by the product            designer. Ability to define a test that combines a set of            predefined (static) items with a set of system-selected            items based on item selection rules defined by the product            designer

Many different types of delivery modes can also be supported, such asPractice, Test Simulation, or Examination modes. Additional configurablefeatures include timing on/off, timing definition, feedback on/off,explanation on/off, ability to return to previous item on/off, testsuspend/resume on/off. Delivery modes are assigned to each test and canbe mixed and matched. Multiple takings of a given test is supported,with performance and history tracked and reported for each taking.

Performance Calculations (Scoring)

Performance calculations allow a student's responses on an evaluatedExercise Component or Test Question to be translated into one or morescores. A score may be used for student self-monitoring, an officialcertification, or for estimation of potential performance on an actualtest. For example, in a continuing education course, a student's finalexam score may be compared to a predefined passing criterion todetermine if certification should be issued.

One simple performance calculation is a raw score expressed as thepercentage of correct responses divided by the total number of questionsin a test. More complex performance calculations involve penaltycalculations and scaling conversions.

The testing system can provide a logic based assessment system that isbased on a computer assisted feedback (CAF) system, such as the currentKaplan Computer Assisted Feedback System. The CAF system can be used intest preparation education centers to assess paper and pencil testsadministered in the centers. The online system administers the testsonline or allows the student to input the answers on paper-based testsusing an online score sheet user interface.

Some examples of CAF logic tests are shown in Appendix A, as Table A.2.In these examples, a test is preformed on a given number of test itemsand the criteria for determining which diagnostic outcome to recommendis based on determining if the test is true for the greater number ofitems in the set, as opposed to an equal number or a number less thanthe specific criteria. Ways of changing the diagnostic strategy are tochange the number set or change the default consideration from “greaterthan some number” to “equal to” or “less than”. These tests generallyassume that questions are numbered consecutively throughout the test(e.g., Section 2 begins with 31, not 1).

Assessment feedback can be based on a series of logic tests that providea significant degree of individualized assessment of students' strengthsand weaknesses as assessed from a diagnostic test or a combination of adiagnostic test and questions from the student profile. The assessmentrules are used by the platform to deliver both 1) an individualdiagnostic reports package for a customized student report and/or 2) therecommendation of learning components of an individual prescriptivestudy plan.

Reporting

The OLTP delivers individual student reports from within a singlespecific product, but other variations are possible. A student report isan expression of performance on an evaluated component, presented in aformat easily understood by the student. The reports componentencompasses the data and methods used to produce thisstudent-interpretable information.

Student reports for a course may be the standard reports, such as thoseproviding percentage scores for tests and categories and item analysisfor correct and incorrect responses or more sophisticated reports. Adiagnostic reporting process (DRP), which might be part of reportingsystem 66 illustrated in FIG. 1C, provides information on a student'sperformance on a diagnostic test in specific categories that can be usedby the student to identify content strengths and weaknesses inparticular content areas. The greater level of detailed reportingprovided by the DRP may be based on a diagnostic test and or studentprofile information. A DRP provides the student with a multi-page DRPthat contains very specific information that may include a narrativestudy plan that illustrates a course of study through products.

Agent reports, such as class aggregate reports for principals andteachers, are provided to institutional settings through the integrationof the OLTP and other management systems. Reports can be used as onlineassessment tools and provide navigation between and among a variety ofdata elements using a browser. Reports can include single test reportingand aggregate test reporting, complete test history (e.g., answerselection history, time per questions, performance), CAF results ineither programmatic form or image/printable form.

Some sample report types will now be described. The exemplary reportsfall into two general types: descriptive and interpretive. A descriptivereport provides data detailing performance on one or more evaluatedcomponents. The data is typically expressed in numerical and graphicformat and may be accompanied by nonvariable explanatory text.

Descriptive reports might differ in the scope and nature of datapresented. For example a discrete report presents data for a singleentity, such as the results for an individual test-taking orlesson-taking. A discrete report allows the student to scrutinizeperformance on the reported taking in isolation from other takings. Sucha report might include question details in an item-level reportassociated with a discrete report. Question details provide the studentaccess to individual questions with the correct answers and thestudent's answers indicated as well as any associated metadata, such asmarkings. Another such report is an aggregate report, which presentscumulative data for multiple entities of the same type, such as aperformance in a category across a group of tests. An aggregate reportallows the student to examine cumulative performance across entities.

A comparative report presents data for multiple entities of the sametype, such as a set of diagnostic tests. The data is presented in amanner intended to facilitate comparisons across the reported entities.A comparative report may contain both discrete and aggregate data.

Interpretive reports interpolate data with performance-specificmessages. Examples of reports are listed in Table A.3(a) in Appendix A.Examples of a Diagnostic Report Package is shown as Table A.3(b) inAppendix A. A Diagnostic Report Package (DRP) is a set of materialsintended to provide a reflection of a student's current performancelevel in a content area and concrete suggestions for improvement. A DRPcan be generated by OLTP 60 processing data from one or more diagnosticmeasures, such as a diagnostic test or a questionnaire. A DRP can alsomap to instructional content that is offline (e.g., print-based), online(within the course producing the DRP), or a hybrid of offline andonline. A DRP often has one or more of the elements shown in TableA.3(b).

Curriculum Delivery System

Overview: A course in the Online Learning Platform is defined in termsof which units, lessons and/or tests are included in the course StudyPlan. Course components could include: study plans, units, lessons,tests, tutorials, references tools, reporting, academic support, andhelp. Unit content may vary in terms of which lessons, tests andreference tools are included within the unit. Lessons and tests may varyin terms of (1) the number of included lesson or question items and (2)which type of lesson and question items are included. Student reportsare either standard statistical analysis or rich assessment feedbackreports, which can include narrative descriptions of a recommendedcourse of study. In addition courses may contain supplement componentssuch as references and tools. The OLTP can support an internal contextsensitive glossary and link to a flashcard tool.

A basic tutorial products category supports the delivery of simple andcomplex lessons on a standalone basis or with the integration of testcomponents as defined above.

A prescriptive learning product category includes a collection ofcomponents as well as rules for (1) prescriptive content delivery for acustom study plan, or (2) product customization based on properties suchas geographic location or instructional agency as criteria fordetermining content and navigation parameters of a course. The systemgathers student profile preferences from the end-users via a websiteand/or enrolled data and/or uses information from diagnostic assessmentsto deliver a customized study plan and a unique learning experience to astudent.

An OLTP inference process applies a product designer's rules to studentdata to produce an individualized study plan to address the student'sspecific learning needs. Individualization may occur by a) providing aset of recommended components, b) changing the strength ofrecommendations for a set of components or c) a combination of both. Therules for recommending instructional lessons, tests and supplementalmaterials can be inputted into the prescriptive instruction systemthrough the CMS and CPS.

The study plan can be provided up front as a student starts to use abody of instructional material, such as via a main menu. The study planoffers the scope and sequence of “plannable” components that mayaccessed by students as part of an online curriculum experience. Theplannable components might include components identified as Units,Lessons and Tests.

When developing a course in the system, the instructional designer wouldplan the set of course materials for a given enrollment and determinethe course control strategies that will be applied to the plan-ablecomponents. The study plan can be generated and viewed within a localonline system or remotely, such as over the Web.

Study plans can contain any type of plan-able component (i.e., Units,Lessons, Tests, and Custom Test Factories) that are contained within theOLTP, as well as links to PDF files served by the OLTP, links tothird-party, stand-alone applications (e.g., Flash Flashcards) and/orunlinked text (e.g., an instruction to do an offline activity). A studyplan might include information pertaining to recommendation levels, datelast accessed, score, status, progress, etc., where some of the elementsare calculated values (e.g., for third-party stand alone applications orfor third party websites), for each plan-able component of the studyplan.

Unit and Lesson Structure

A Unit is an aggregation of Lesson and/or Tests components in a definedgrouping. A Lesson is a predefined sequence of instructional deliverableitems addressing one or more closely related learning. Eachinstructional deliverable item, also known as a Lesson Item, isdeveloped to support, or evaluate, a single learning objective. TheInstructional Designer can support the teaching of the learningobjective using as many Lesson Items as they desire. The OLTP cansupports Lesson Item types such as Instruction, Activity, Exercise andSupplement.

Lesson Item Types

1. Instruction Items require no explicit user interaction and apply toitems such as text, reading passages, static graphics, animatedgraphics, or links to other Lesson Items, content-sensitive content,downloadable objects and the like.

2. Activity Items include user interaction that is not evaluated and nottracked by the system, such as self-contained experiential elements,text or instructions to perform offline activity, or animated graphicswith user controls such as manipulated elements.

3. Exercise Items include student-response data recorded by the OLTP,immediate evaluation items, correct/incorrect response messages,explanations (may be provided by system or at the student's request),hints (may be provided by system or at the student's request), and thelike. Response types can be optional, required or under course control,a response contributed to lesson completion and performance information.Some exercise items are gated, in that a correct response is requiredbefore proceeding to the next item in a lesson sequence, such as forverifying comprehension. Exercise items might also include responsesupport, where a hint or explanation is provided after an incorrectanswer.

4. A Supplemental Item might be an optional Lesson Item or sequence ofLesson Items to extend or review a concept and might be limited to useonly when some students need additional information, preferably notincluding exercise items. A Lesson may have zero or more links toSupplemental Items

Course Control

The OLTP provides a unique set of rules that provide course controlswithin a Unit, Lesson or Course. The course controls allow aninstructional designer to structure students' paths through coursecontent. Course control can be access control, achievement control, or acombination thereof. For access control, preconditions need to be metbefore allowing access to a component and constraints can be placed onhow many times a component may be repeated. For achievement control, astudent stays on a component until conditions are met such that thecomponent is considered finished or until comparisons between studentperformance and specified benchmark criteria indicate completion. Coursecontrols are optional and may be used in combination, thus providinggreat flexibility in supporting variations in course designs.

Authoring Example

FIG. 9 is a sequence of screen shots (FIGS. 9A-9E) illustrating aprocess of authoring content. One pass through an authoring session isshown in FIGS. 9A-9E. Each of these figures is a simplified screen shotof an exemplary application.

The authoring tools provide a software environment where authors createquestion and lesson content. The tools can automatically andtransparently encode the content with XML tags to provide compatibilityand consistency with the CMS data model. Support of content creationincludes producing the associated files, including the productItem filesand productDeliveryRules files described below, as well as the item andcontent files. A “productItem” is represented by an XML file withmetadata describing the instructional content's pedagogic and reportingcategorization within a course; productItem also might contain aone-to-one reference with an XML file containing instructional contentto be presented within the course. A “productDeliveryRules” isrepresented by an XML file containing instructions on how a piece ofinstructional content is delivered and processed within the course. Forexample, a productDeliveryRule determines if a question must be answeredbefore continuing within the course and if a question will be evaluated.

The authoring tools provide authors with the ability to choose betweencreating lesson items and creating test question items. The configurableenvironment allows the author to enter into handle content for atest-specific area (such as GRE, GMAT, SAT, etc.) and use global andspecific text and structural formatting types configured for specificquestion types of that test. Authors can create templates for specificpresentation layouts for lessons.

The authoring tools include presentation tools, such as tools forspecifying format, such as text formatting using predefined emphasistypes and XHTML, visually formatting text, inserting special charactersand symbols, text copy, cut and paste, etc. The authoring tools alsoinclude tools for inserting inline and/or stand-alone media referencesinto content either by browsing/searching a repository for preexistingmedia items or by allowing the author to add media items at time ofcontent creation.

Using the authoring tools, an author can insert and apply layout-relatedformatting (e.g., bulleted lists, test question stem/choices), enterquestion item sets in a continuous setting (vs. individual questionitems), locate all content types (e.g., questions, lesson pages, staticmedia, rich media) within the repository by searching on associatedmetadata, preview content page layout prior to publishing of thecomplete product to the OLTP and lay out a course structure by arranginga sequence of pages into units, lessons and tests. The authoring toolsalso allow authors to communicate to product assembly, the structure ofa course as well as content files included in the course'sunits/lessons/tests.

As shown in FIG. 9A, an author indicates that a new file is to bestarted for a lesson and selects a type for the new file; “Lesson Page”in this example. Other file types might include Lesson Page, TestQuestion Item, Test Tutorial Item, etc. As shown in FIG. 9B, the authorcan then type in text associated with the file and apply formatting. Asshown in FIG. 9C, the author can add other structures to the file, suchas images, rich media, side bars (e.g., side bar 310), tip bars, etc.Some structures might have substructures, such as side bar 310 having aheader area and a content area where the author can insert separate textand possibly other data. Another example is tip bar 312 shown in FIG.9D.

In addition to text, the author can insert images or other objects, asshown in FIG. 9D, with options to align the objects to the text invarious ways (e.g., left, right, centered). Text can be formatted usinga format menu or using icons. Links can also be added to the text, suchas by including a URL as part of an anchor. Once the author enters theremaining text of the lesson, the author can add metadata for the file,as illustrated in FIG. 9E and save the file or perform other actions.

Thus, although the invention has been described with respect toexemplary embodiments, it will be appreciated that the invention isintended to cover all modifications and equivalents within the scope ofthe following claims. TABLE A.1. Appendix A. Sample Product DefinitionParameters (PDP) Table Parameter Description Options I. FunctionalityProduct Class The name of the product class that Custom Test/Drill andthe course is defined by Practice/Continuing Ed/ Prelicensing ProductLine The name of the product name that NAPLEX the course is defined by.This may correlate to a specific look and feel, e.g. GMAT ProductRelease Version The version number for the product 1.0.0 Minimum KLPVersion The minimum version of the KLP R2 that the product is expectedto normally function on Course Definition Plannable Component List Listof plannable components to be Custom Tests included in the course.Plannable course components are: Units, Lessons, Predefined Tests,Custom Tests Study Plan Display Order List of plannable components thatCustom Tests will be included in the Study Plan in the desired displaysequence Course Completion Criteria The rules that determine whether N/AStrategy the course has been completed by the user Course PassingCriteria The rules that determine whether Not Applicable Strategy thecourse has been passed by the user Category Definition Data valuescomprising the No Categories categories Difficulty Level Scale Theinternal numerical scale 0 representing the range of Difficulty LevelsDifficulty Level UI Mapping Mapping of Difficulty Level UI 0 = N/Apresentation (e.g. 1, 2, 3) to internal numerical scale representationItem Flag Number The number of Item Flags to be 0/1/2 included Item FlagLabels Text values for one or both Item “Guess” Flags Specify for eachplannable course component: Plannable Component Display Name of theplannable component NAPLEX Quiz Bank Name that is to be displayed in theUI Plannable Component Type The plannable component type Specify thetype: Unit/ Lesson/System-Generated Test/Custom Test Plannable ComponentThe plannable component NAPLEX Quiz Bank Classification classificationas a “Final Exam” or other instructional construct. There are no pre-setclassifications; they are fully definable by the product designer.Recommendation Level Whether the plannable course Required/Optionalcomponent is required or optional. For R2, this data is for displaypurposes only (versus course control). Plannable Component The rulesthat determine whether a Select rule(s): Final Completion CriteriaStrategy plannable component has been exam taken/Final exam completed bythe user passed/All lesson materials accessed/ Specified amount of timespent in lesson content/ Unit posttest taken/Unit posttest passed/Additional rules: All interactive lesson items completed with correctresponses on last attempt/ Last question answered (if reverse navigationis not permitted)/Time limit reached/Student invoked exit PlannableComponent Passing The rules that determine whether a Select strategy:TBD Criteria Strategy plannable component has been passed by the userPlannable Component Scoring The rules that determine how a Selectstrategy: TBD Strategy plannable component should be scored PlannableComponent The rules that determine how a Select strategy: TBDTermination Strategy plannable component may be terminated PlannableComponent Category The category(ies) assigned to the Provide categoryvalues Value specific plannable component from the set of valuesassigned to the course overall Plannable Component Difficulty TheDifficulty Level assigned to the Provide Difficulty Level Level specificplannable component values from the range of values defined for thecourse overall Item Selection Strategy The rules for selecting items forthe Select: Random/ Plannable Component, in the case Predefined of Testsand Lessons Delivery Mode The Delivery Mode that a Test or Select Mode:Test Lesson should be presented in simulation/Practice/ ExaminationDelivery Modes (For each Delivery Mode, specify the following) TimingMode Whether a test is untimed or timed. Untimed/System A test may bedefined as timed by Selected Timed/Student the product designer or theoption Selected Timed may be provided to the user to take the test in atimed mode. Timing Method Whether timing occurs at the Select method:Plannable Component, Section or Plannable Component/ Selectable Itemlevel Section/Selectable Item Timing Limit Time limit for an independent80 element or a sum of elements depending on the value of Timing MethodTest Suspend Inclusion Inclusion of the ability to suspend a Include/Donot include test Answer Confirm Inclusion Inclusion of the presence ofan Include/Do not include Answer Confirm button in the UI Previous ItemNavigation Whether to include the ability to Include/Do not includeInclusion navigate to the Previous Item Response Evaluation MessageWhether the Response Evaluation Include/Do not include Inclusion Messagefeature (e.g. Your answer is correct/incorrect) should be included inthe UI Explanation Inclusion Whether the Explanation feature Include/Donot include should be included in the UI Explanation Link DisplayDefines where the Explanation link With Response should be presentedEvaluation Message/ External to Response Evaluation Messages/Both ItemFlag Inclusion Whether the Item Flag feature Include/Do not includeshould be included in the UI Item Review Inclusion Whether to includethe ability to Include/Do not include access Test Item Review LessonDisplay Mode Whether the Test or Lesson UI is Component Region/Pop-updisplayed within the Component Region or as a separate pop-up windowFlashcards Access Whether to allow access to Access/No access Flashcardswhile taking a test or lesson Tips Access Whether to allow access toTips Access/No access while taking a test or lesson Reports AccessWhether to allow access to Reports Access/No access while taking a testor lesson Question Report Access Whether or not to permit the Access/Noaccess Student to access/view the Question Report (which provide answerdetails) while taking a Test or Lesson, if access to Reports isavailable Glossary Access Whether to allow access to Access/No accessGlossary while taking a test or lesson Help Access Whether to allowaccess to Help Access/No access while taking a test or lesson GeneralTest Parameters (For each Test, specify the following) Total Number ofItems in Test The total number of items included 185 in a test Test ItemList List of items that may be included in Determined by Business atest. This may be a preordained, Unit nonvariable list of items or a setof items from which a given test may be generated. If a semblance ofweighting by category and/or difficulty level is desired, the listshould be Test Mode Instructions Provide option for the user to skip Mayskip/May not skip Skippable test mode instructions (I.e. practice,simulation, examination mode) Target Test Instructions Provide optionfor the user to skip May skip/May not skip Skippable target testinstructions (I.e. test emulation) Custom Test (For each Custom Test,specify the following) Number of Items Allowed The maximum number ofitems 185 allowable for a custom test Difficulty Level UI InclusionInclusion of Difficulty Level in UI Include/Do not include versus initem data Reuse UI Inclusion Inclusion of Reuse Heuristic Include/Do notinclude Reuse Values Definition of reuse heuristic values All/NotUsed/Incorrect Only/Incorrect and Not Used Default Test Name The namethat will be offered to the Test 1 Student at point of test creation. Itmay be overridden by the Student Lessons (For each lesson, specify thefollowing) Sequence of Instructional Items A list of the instructionalitem N/A identifiers in the order that the items will be displayedSupplemental Items A list of linked supplemental items N/A representedin the order that they will be displayed Item Set (For each item set,specify the following) Performance Calculation The rules for calculatingPercent Correct Strategy performance on an item Next Item Strategy Therules for determining which Sequential item to present next ItemSelection Strategy The rules for selecting selectable Select:Random/Predefined items for the item set Item Set Time Limit The maximumamount of time Provide an integer in allowable for the Student to selectmilliseconds an answer choice Shuffle Enabled Whether a items should beshuffled Yes/No in the case of a new taking of a system-generated testSelectable Item (For each selectable item, specify the following)Shuffle Override If shuffling is enabled, the ability to Yes/No preventthe shuffling of selectable items, e.g. in the case of ReadingComprehensive items that build upon each other Selectable Item CategoryValue The category(ies) assigned to the See Category XML doc selectableitem Selectable Item Difficulty Level The Difficulty Level assigned tothe 0 selectable item Required Items List of items that MUST be includednone in the test, if any Deliverable Item (For each deliverable item,specify the following) Question ID (QID) The Customer Service or VendorSpecify the identifier number associated with the item, (Definable byBusiness e.g. typically used by Customer Units) Service to reference anitem that the Student is having a problem with Deliverable Item CategoryThe category(ies) assigned to the See Category Doc Value deliverableitem (Definable by Business Units) Deliverable Item Difficulty Level TheDifficulty Level assigned to the 0 deliverable item Response ExpectedWhether a response to an item is Yes/No expected, e.g. in the case oftest items and lesson activities and exercises Response Scorable Whethera response should be Yes/No scored e.g. in the case of test items andlesson exercises Response Evaluatable Whether a response should beYes/No evaluated e.g. in the case of lesson activities Intro Sequenceand Navigation Parameters Orientation Inclusion Inclusion of OrientationInclude/Do not include Orientation Skippable If included, provide optionfor user May skip/May not skip to skip the Orientation Reporting ReportClassification Display The order of Plannable Component Define the orderof Order Classifications by which reports will Plannable Component bedisplayed Classifications References Glossary Inclusion Inclusion ofGlossary Include/Do not include Help and Support Help InclusionInclusion of Help Include/Do not include Academic Support Inclusion ofAcademic Support Include/Do not include Technical Support Inclusion ofTechnical Support Include/Do not include II. Look and Feel Look and FeelStyle Template Choice of the UI template that will KTP Grad/KTP K12/comprise both the top horizontal KTP USMLE/Financial/ branding andnavigation region Real Estate/Custom (product region) and the contentregion (component region) Logos (Images provided and/or selectable bybusiness units) Product Name Logo Provide graphics for inclusion in theWelcome screen and branding region. Business Unit/Product Group Providegraphics for inclusion in the Welcome screen and branding region.Co-Branding Partners Provide graphics for inclusion in the Welcomescreen and branding region. III. Component Region UI Test InterfaceChoice of the Test (test taking and Best Practices/ETS/NASD item review)component UI which may be either the Best Practices Test UI or astandardized test format UI. IV. UI Variable Copy (For Options, useVariable Copy Doc options (Definable by Business Unit)) Welcome PageProduct Name Name of the product in text format (versus graphic)Publisher Name(s) Copyright Copyright language Trademark Trademarklanguage Salutation Salutation to the first time user, e.g. “Hi” or“Hello” Return Salutation Salutation to the returning user, e.g.“Welcome back” Welcome Message Welcome message to the first time userWelcome Back Message Welcome back message to the returning user LearningObjectives Statement of course learning objectives Orientation PageOrientation Product orientation message Test Test Directions-TestSimulation The instructions presented to the Mode user before entering atest in Test Simulation mode Test Directions--Practice Mode Theinstructions presented to the user before entering a test in Practicemode Test Directions-Examination The instructions presented to the Modeuser before entering a test in Examination mode Test Directions -Pre-defined The instructions presented to the Test user before enteringa pre-defined test Standard Test Format The instructions presented tothe Directions user before entering a test presented in a StandardizedTest Format UI Study Plan Course Objectives Course objectives presentedon the parent Study Plan page Other Help Help copy that is custom to thecourse

TABLE A.2. Appendix A. CAF Logic Tests Code Title Description TableA.2(a) Classic Logic Trees QC Quantity Out of the cited questions, ifmore Correct than a certain number are correct, the test returns TRUE.e.g., 4, 3, 5, 9, 10, 15 The first number is the “certain number.” Ifgreater than 4 of questions 3, 5, 9, 10, 15 are correct, the testreturns true. QS Quantity Out of the cited questions and a.c.s.,Specific if more than a certain number are the student's responses, thetest returns TRUE. e.g., 4, 3A, 3C, 9_, 10C, 15D The first number is the“certain number.” If greater than 4 of questions of the followingresponses were given by the student, the test would return true:Question 3, choice A or C; Q9 left blank; Q10 choice C; Q15 choice D. QOQuantity Out of the cited questions, if more Omitted than a certainnumber were omitted, the test returns TRUE. e.g., 4, 3, 5, 9, 10, 15 Thefirst number is the “certain number.” If greater than 4 of questions 3,5, 9, 10, 15 were omitted, the test returns true. BL Blanks If there isone or more blank on the entire test, BL returns TRUE. FS Final ScoreThis test looks at what is stored in the “Final Score” field of thescore history database and sees if it is greater than a certain number.e.g., 129 Is the score greater than a 120 (say, on the LSAT)? TableA.2(b) Advanced Logic Trees MS Macro Interprets and evaluates logicalSubstitution expression with variables. Can use the “sorter” functionallowing comparison of values. e.g. score (1, 1) > score (2, 1) + 40score (1, 1) = Quantitative scaled score on this test score (2, 1) =Quantitative scaled score on previous test (by date) If Q score on thistest is 40 points higher than previous Q score, then the test returnstrue. VA Variable Can create a variable for subsequent Assignment logictests. This test is not evaluated as true or false. e.g. weakscore =sorter(“3; score(1, 1); score(1, 2); score (1, 3); 1”)* The sorterfunction returns the xth lowest value. X is the last number of thefunction. Here, we are looking for the lowest scaled score (Q, V, or A)on the current test. Rather than repeatedly having to call the function(wasting processing time), we can test for the value of weakscore. LPLesson/Study A 2-digit character string representing Plan one of the 45Study Plansn(GMAT) is determined elsewhere in the program (e.g. “13” or“07”). When this Logic Tree type is invoked, it just prints whicheverLesson Plan was selected for this student and then goes on to the nextlogic tree (via the “goto true” field). The PICT number for Lesson/StudyPlans is “90” + the 2-digits representing the study plan. While StudyPlans can be determined within the logic tree structure, time-consumingor complicated logic designs are handled within the program. It is onlythe latter case that necessitates the LP Logic type. Table A.2(c)Functions * Sorter This function is used in the advanced logic treetypes. It returns the name of the variable (uppercase) that holds aparticular rank among a specific set of variables. Numeric and charactervariable values are sorted in ascending order. You can then look for thevariable that holds a specified place in the ordering of the values. Thesyntax: sorter(“x; var1; var2; . . . varx; n”) Where x is the number ofvariables being considered, var1 . . . are the names of the variables(or array elements), and n is the nth lowest value which you are lookingfor. For example, sorter(“3; score(1, 1); score(1, 2); score(1, 3); 1”)will return the (1st) lowest scaled score attained on the current test.If the student scores were 450Q (score(1, 1)), 370V (score (1, 2)), 580A(score (1, 3)), then the sorter function would return SCORE (1, 2). STSAY TEXT Prints whatever text occurs in criteria field. SV SAY VARIABLEPrints the value of the variable occurring in criteria field.

TABLE A.3. Appendix A. Example Reports Name Description Data TableA.3(a) Specific Reports Individual List of all student- For each test:Tests Main created tests number correct number attempted percentagecorrect status Individual Overview of an For selected test: Test Summaryindividual test number correct number attempted MARK_1 (if used) MARK_2(if used) total changed number changed incorrect to correct numberchanged correct to incorrect number changed incorrect to correctcategory performance summaries Question List of all items for For eachtest item Details an individual test in selected test: (sub-report ofsequence number Individual Test unique identifier Summary) whethercorrect, incorrect, or incomplete associated category name how changedhow marked difficulty Category Overview of performance For each categoryin Details in each category tested selected test: on an individual testcategory name (sub-report of number correct Individual Test Summary)number attempted percentage correct Category List of performance in Foreach category Summaries each category across all tests: summarizedacross category name all tests number correct number attemptedpercentage correct Category List of all tests For each test: by Testshowing performance for number correct the selected category numberattempted within each test percentage correct (sub-report of CategorySummaries) Question Overview of performance Of all items: Summary acrossall items total available number attempted number correct on firstattempt number correct on most recent attempt total changed numberchanged incorrect to correct number changed correct to incorrect numberchanged incorrect to incorrect Lesson List of all lessons For eachlesson: Reports categories (if applicable) number correct on firstattempt number correct on most recent attempt number possible(attempted) percentage correct (if applicable) repetitions [criterion-For each test (in referenced addition to data for tests] R1 TestSummary): passing criterion

TABLE A.3(b) Diagnostic Report Package Element Purpose ApplicationDescriptive Statistics Display numeric and graphic any product(Analysis) results of one or more diagnostic tests Narrative MessagesRelay nonvariable and/or any product (Diagnostic Profile, variabletext-based Diagnostic Feedback) information related to performance ondiagnostic measures Question Details Display student's responses anyproduct with (Report Answer to test questions with one or more Review)indication of whether online diagnostic responses were correct, testsincorrect, or omitted Response Summary Display both correct and productwithout (includes an student's responses to an online answer key) testquestions; summarize diagnostic test some question performanceinformation Study Time Allocation Prioritize study topics productwithout (Study Plan Summary, and allocate blocks of online instructiontime budget) time to each topic Offline Study Plan Prescribe an offlineproduct without course of study online instruction (or a hybrid ofonline and offline instruction)

1. (canceled)
 2. An online curriculum handling system, wherein aplurality of students interact with curricula supported by the onlinecurriculum handling system using computers, the online curriculumhandling system comprising: a content management system, wherein atomiccontent components are stored independently of product content; astudent profile database; a database of product assembly templates; aproduct publishing system including means for constructing an onlinecurriculum product for use by the plurality of students with referencesto atomic content components stored in the content management systemusing at least one product assembly template from the database ofproduct assembly templates, wherein a product indicated as customizablecan be automatically customized for a given student from the informationstored in the student profile database for the given student; and afeedback module for updating student profiles in the student profiledatabase in response to student activity with respect to publishedproducts.
 3. The online curriculum handling system of claim 2, whereinthe student profile database includes target skill sets, assessmentlevels and prior test performance for at least some of the plurality ofstudents.
 4. The online curriculum handling system of claim 2, whereinthe atomic content components are organized according to a taxonomy. 5.(canceled)
 6. (canceled)